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Abstract 


Workforce  effectiveness  relies  on  two  critical  characteristics:  competence  and  readiness.  This 
technical  note  describes  the  Competency  Lifecycle  Roadmap  (CLR),  a  preliminary  roadmap  for 
understanding  and  building  workforce  readiness,  developed  by  the  Computer  Security  Incident 
Response  Team  (CSIRT)  Development  and  Training  team  at  the  CERT  ”  Program,  part  of 
Carnegie  Mellon”  University’s  Software  Engineering  Institute.  This  note  provides  an  early  look  at 
the  roadmap,  highlights  some  of  its  uses  to  date,  and  discusses  potential  next  steps  in  its 
development  and  transition. 
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1  Introduction 


Workforce  effectiveness  relies  on  two  critical  characteristics:  competence  and  readiness. 
Competence  is  the  sufficient  mastery  of  the  knowledge,  skills,  and  abilities — or  competencies — 
needed  to  perform  a  given  task.1  Competence  reflects  how  well  an  individual  understands  a 
subject  matter  or  is  able  to  apply  a  given  skill,  and  it  is  necessary,  but  not  sufficient,  to 
successfully  perform  a  given  task.  Readiness  is  the  ability  to  apply  a  set  of  competencies  to 
complete  a  real-world  task. 

Consider  the  following  scenario:  A  large  agency  recently  hired  several  people  to  join  its  digital 
analytics  team.  The  new  employees’  long-term  job  was  to  perform  forensic  evidence  collection 
and  subsequent  digital  media  analysis  in  the  field  and  back  at  the  organization’s  test  lab.  The  new 
hires  were  put  through  initial  training  to  teach  them  how  to  perform  these  tasks,  including  using 
various  forensic  tools  and  associated  analysis  processes.  Tools  that  were  reviewed  included 
applications  such  as  EnCase2  and  FTK.3 

After  the  new  employees  completed  training  for  each  technique  or  tool,  they  were  tested  to 
determine  if  they  had  acquired  the  knowledge,  skills,  and  abilities  needed  to  use  it.  The  new  team 
members  successfully  passed  all  of  the  individual  tests  presented  to  them.  After  they  completed 
all  of  their  introductory  training  courses,  the  team’s  leader  believed  that  the  new  team  members 
were  ready  for  field  work.  However,  when  the  new  team  members  were  presented  with  a 
compromised  system  to  analyze  in  the  field,  none  of  them  were  able  to  perform  the  analysis 
adequately.  They  had  no  idea  how  to  start  the  analysis  and  investigation  process  on  their  own. 
They  were  also  unfamiliar  with  the  media  and  how  to  collect  data  from  it.  Although  they  could 
perform  parts  of  the  process  and  use  various  tools  in  a  classroom  setting,  they  failed  the  test  that 
mattered  most.  They  could  not  adapt  what  they  had  learned  to  the  environment  of  the  real-world 
scenario.  In  the  end,  the  new  team  members  were  not  ready  to  perform  their  job  function  in  the 
field. 

In  this  scenario,  the  new  team  members  demonstrated  their  competence  with  the  suite  of 
techniques  and  tools  needed  for  their  job  assignments.  However,  they  were  unable  to  analyze  a 
compromised  system  in  the  field.  They  lacked  the  readiness  to  perform  their  assigned  tasks  in 
real-world  conditions  despite  having  shown  they  possessed  the  required  knowledge,  skills,  and 
abilities. 

This  scenario  showcases  a  common  problem  experienced  by  organizations  with  training  and 
development  programs  that  validate  only  specific  skills  learned  in  isolation.  Without 
understanding  what  true  readiness  entails  and  testing  a  staffs  ability  to  accomplish  those 
requirements,  organizations  often  come  up  short  of  achieving  their  true  training  and  development 


For  discussion  of  competencies,  see  The  People  Capability  Maturity  Model  -  Guidelines  for  Improving  the 
Workforce  [Curtis  2002]  and  Project  Manager  Competency  Development  Framework  [PMI  2002], 

http://www.guidancesoftware.com/encase-forensic.htm 

http://accessdata.com/products/computer-forensics/ftk 
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goals.  Staff  must  be  able  to  show  that  they  can  apply  what  they  have  learned,  and  training 
programs  must  be  structured  to  provide  staff  with  opportunities  to  test  that  readiness  before  being 
placed  in  the  field  and  confronting  unexpected  situations. 

1.1  Competency  Lifecycle  Roadmap  (CLR) 

Researchers  from  the  CERT1'  Program  at  Carnegie  Mellon”  University’s  Software  Engineering 
Institute  (SEI)  created  the  Competency  Lifecycle  Roadmap  (CLR),  which  defines  a  systematic 
approach  for  developing  and  sustaining  workforce  readiness  over  time.  We  designed  the  CLR  to 
be  domain  independent  so  that  it  can  be  broadly  applied  across  multiple  disciplines.  It  takes  into 
account  a  vast  body  of  knowledge  in  the  cognition  and  perfonnance  disciplines  pioneered  at 
Carnegie  Mellon  University,  including  research  into  the  nature  of  expertise  [Simon  1996].  This 
roadmap  comprises  five  core  activities  (assess,  plan,  acquire,  validate,  and  test  readiness)  and  two 
foundational  elements  that  support  those  activities  (criteria,  environment).  Section  2  of  this 
technical  note  details  the  roadmap  and  its  activities  and  foundational  elements. 

We  envision  using  the  CLR  in  many  ways,  including 

•  identifying  gaps  in  an  organization’s  training  and  development  program  by  benchmarking  it 
against  the  CLR 

•  helping  an  organization  apply  the  CLR  to  improve  its  training  and  development  program 

•  providing  guidance  for  developing  curricula  or  training  programs 

•  helping  an  individual  set  personal  goals  related  to  a  specific  job  or  task 

1.2  Background 

For  the  past  several  years,  the  CERT  Program  has  helped  client  organizations  improve  their 
training  and  development  programs.  Several  of  these  engagements  focused  on  identifying  and 
documenting  cybersecurity  competencies.  However,  as  in  the  scenario,  organizations  began  to 
understand  that  competence  is  not  readiness. 

In  2011,  the  CERT  Program’s  Computer  Security  Incident  Response  Team  (CSIRT)  Development 
and  Training  (CDT)  team  chartered  a  project  focused  on  building  readiness  within  an 
organization’s  workforce.  We  developed  the  workforce  readiness  project  based  on  lessons  learned 
from  identifying  and  documenting  cybersecurity  competencies  for  client  organizations,  as  well  as 
observations  of  work  perfonned  in  related  software  engineering  settings.  The  project’s 
competency  work  initially  focused  solely  on  cybersecurity.  However,  over  time  the  project  scope 
broadened  to  include,  for  example,  supervisors’  readiness  to  perfonn  their  assigned  leadership 
duties. 

As  we  shifted  the  focus  of  our  research  from  developing  and  documenting  competencies  to 
building  workforce  readiness,  we  leveraged  previous  SEI  work  in  building  readiness  using 
certification  programs  [Behrens  2004]  and  cybersecurity  workforce  development  [Hammerstein 
2010],  We  also  looked  at  relevant  research  throughout  the  training  and  development  community. 
In  particular,  we  reviewed  materials  focused  on  how  to  build  competency-based  training 


®  CERT  and  Carnegie  Mellon  are  registered  marks  owned  by  Carnegie  Mellon  University. 
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programs,  evaluation  and  assessment  methods,  and  methods  for  developing  effective  training  and 
development  programs.  The  Bibliography  of  this  technical  note  lists  these  materials. 

While  the  SEI  has  significant  experience  in  defining  and  documenting  cybersecurity  competencies 
for  its  customers,  this  work  is  not  unique  to  the  SEI.  Over  the  past  few  decades,  organizations 
throughout  the  community  have  undertaken  many  similar  efforts  to  develop  and  document  lists  of 
competencies.  For  example,  both  the  U.S.  Office  of  Personnel  Management  (OPM)  and  the 
National  Initiative  for  Cybersecurity  Education  (NICE)  have  developed  and  documented 
competency  frameworks  for  the  cybersecurity  community  [OPM  2011,  NICE  2011].  The  OPM 
has  also  developed  and  documented  a  Leadership  Competency  Framework  that  focuses  on  an 
organization’s  management  and  leadership  roles  [OPM  2006]. 

1 .3  About  This  Technical  Note 

The  primary  audience  for  this  technical  note  is  managers  and  training  officers  who  want  to 
improve  their  organization’s  training  and  development  program.  Researchers  focusing  on  training 
and  education  activities  will  also  find  this  document  useful.  It  will  also  benefit  individuals  or 
small  working  groups  trying  to  develop  their  own  competence  and  readiness  to  perfonn  or 
improve  their  work. 

In  general,  people  who  are  interested  in  the  following  topics  will  find  this  technical  note 
worthwhile: 

•  competency  definition  and  development 

•  building,  benchmarking,  and  improving  a  training  and  development  program 

•  understanding  personal  (or  team)  goals  for  competency  development  and  readiness 
improvement 

The  overarching  goal  of  this  technical  note  is  to  describe  a  preliminary  roadmap  for  understanding 
and  building  workforce  readiness,  though  the  CLR  is  still  early  in  its  development.  This  note 
provides  an  early  look  at  the  roadmap,  highlights  some  of  it  uses  to  date,  and  discusses  potential 
next  steps  in  its  development  and  transition.  Future  reports  will  further  detail  the  roadmap’s 
technical  aspects. 

This  remainder  of  this  note  comprises  the  following  sections: 

•  Section  2:  The  Competency  Lifecycle  Roadmap  (CLR) — describes  the  five  activities  and 
two  foundational  elements  of  the  CLR 

•  Section  3:  Implementation  Approaches — provides  a  basic  overview  of  guidelines  and  tools 
for  implementing  the  CLR 

•  Section  4:  Summary  and  Next  Steps — presents  next  steps  in  the  development  and  transition 
of  the  roadmap 

•  Bibliography — lists  related  publications  used  in  creating  the  CLR  and  this  technical  note 
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2  The  Competency  Lifecycle  Roadmap  (CLR) 


The  Competency  Lifecycle  Roadmap  (CLR)  is  an  approach  for  systematically  building  workforce 
competencies  and  maintaining  them  over  time.  The  roadmap  focuses  on  an  individual’s  readiness 
to  perform  his  or  her  task  assignments.  In  this  context,  readiness  is  the  ability  to  apply  the  total  set 
of  competencies  (both  technical  and  enabling)  required  to  perform  a  task  or  set  of  tasks. 

The  roadmap  comprises  five  core  activities — assess,  plan,  acquire,  validate,  and  test  readiness — 
and  two  foundational  elements  that  support  the  activities — criteria  and  environment.  Figure  1 
illustrates  the  basic  structure  of  the  CLR. 


Assess 

Plan 

Acquire 

Validate 

Test 

Readiness 

Criteria 

Environment 

Figure  1:  Competency  Lifecycle  Roadmap  (CLR)  Structure 


This  section  provides  a  conceptual  overview  of  each  roadmap  activity  and  foundational  element. 

2.1  Activities 

A  roadmap  activity  is  defined  as  a  task  performed  to  achieve  a  specific  training  and  development 
outcome. 

2.1.1  Assess 

The  first  activity,  assess,  is  an  initial  evaluation  of  key  competencies  and  the  ability  to  perform 
those  competencies  in  a  specific  task.  This  activity  should  not  be  confused  with  a  training 
assessment,  which  evaluates  the  extent  to  which  a  training  course  meets  its  objectives.  In  contrast, 
the  roadmap’s  assessment  is  a  performance-based  test  that  includes  measurement  of  an 
individual’s  current  competencies.  It  evaluates  an  individual’s  ability  to  apply  a  stated 
competency,  regardless  of  how  that  competency  is  acquired  (e.g.,  coursework,  experience,  or 
observation).  Because  knowledge  can  be  broad  or  specific,  gradually  or  discretely  acquired, 
relevant  for  long  or  short  periods,  or  retained  or  lost  over  time,  a  baseline  assessment  of  an 
individual’s  current  knowledge  and  abilities  is  essential. 

Assessment  is  important  to  the  roadmap  because  this  activity  defines  a  systematic,  objective,  and 
repeatable  process  for  establishing  a  baseline  of  strengths  and  weaknesses  in  the  specific 
competencies  needed  to  perform  a  specific  task.  These  competencies  are  called  identified 
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competencies  in  this  note.4  Assessment  also  provides  insight  into  which  competencies  need  to  be 
maintained  or  improved  to  achieve  the  desired  performance.  In  addition,  as  an  organization 
assesses  groupings  of  competencies,  it  gains  an  overall  picture  of  an  individual’s  relative  strengths 
and  weaknesses,  which  can  assist  that  individual  with  professional  growth  opportunities.  Table  1 
presents  the  key  characteristics  of  the  assess  activity. 

Table  1:  Characteristics  of  Assess  Activity 

Dimension  Description 

What  •  an  initial  evaluation  of  key  competencies  and  the  ability  to  perform  them  in  a  specific  task 

Why  •  to  identify  a  baseline  of  strengths  and  weaknesses  in  the  key  competencies  needed  to 

perform  a  specific  task 

•  to  apply  a  systematic,  objective,  and  repeatable  process 

•  to  provide  insight  into  how  to  maintain  or  improve  the  performance  of  identified 
competencies 


There  are  many  different  ways  to  perform  the  assessment  process.  Each  organization  will  need  to 
determine  what  works  best  in  its  environment.  Examples  of  methods  that  might  be  used  include 
the  following: 

•  Conduct  a  performance-based  test  that  includes  measurement  of  the  current  state  of  key 
competencies. 

•  Have  individuals  complete  a  skills  inventory  with  supporting  substantiation  showing 
evidence  they  mastered  those  skills.  Substantiation  might  be  course  certificates  or  a 
manager’s  recommendation. 

2.1.2  Plan 

The  next  roadmap  activity,  plan,  defines  an  individual’s  intended  course  of  action  for  maintaining 
or  improving  specific  competencies  that  are  needed  to  perform  a  specific  task  assigmnent.  Table  2 
presents  the  key  characteristics  of  the  plan  activity. 

Table  2:  Characteristics  of  Plan  Activity 

Dimension  Description 

What  •  a  course  of  action  intended  to  maintain  or  improve  identified  competencies 

Why  •  to  specify  an  attainable  path  for  maintaining  or  improving  identified  competencies 

•  to  communicate  the  path  for  maintaining  or  improving  identified  competencies 


The  plan  activity  is  important  because  it  specifies  an  attainable  path  that  an  individual  can  follow 
to  maintain  or  improve  identified  competencies.  Here,  an  individual  determines  which  options  and 
resources  are  available  and  relevant.  Once  the  path,  or  development  plan,  has  been  developed,  the 
individual  documents  it  and  then  disseminates  it  to  all  relevant  stakeholders.  Planning  thus  lays 
the  foundation  for  acquiring  identified  competencies,  which  is  the  next  roadmap  activity. 

Some  examples  of  planning  methods  include  the  following: 

•  Map  strengths  and  weaknesses  to  options  and  resources  provided  within  the  organization  and 
community  to  develop  a  path  for  maintaining  or  improving  identified  competencies.  This 
may  often  take  the  form  of  an  individual  development  plan  (IDP). 

•  Document  and  disseminate  the  path  for  maintaining  or  improving  identified  competencies. 


The  identified  competencies  constitute  the  subset  of  all  key  competencies  that  will  be  addressed. 
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2.1.3  Acquire 


The  acquire  activity  of  the  roadmap  defines  actions  that  will  be  taken  to  obtain  the  knowledge  or 
skills  required  to  maintain  or  improve  identified  competencies.5  Acquisition  of  competencies  is 
important  because  it  enables  an  individual  to  reinforce  strengths  and  address  weaknesses  in  his  or 
her  knowledge  and  abilities.  Table  3  presents  the  key  characteristics  of  the  acquire  activity. 

Table  3:  Characteristics  of  Acquire  Activity 


Dimension 

Description 

What 

•  actions  taken  to  maintain  or  improve  identified  competencies 

Why 

•  to  reinforce  strengths  and  address  weaknesses  in  the  ability  to  perform  a  specific  task 

Depending  on  a  job’s  complexity  and  requirements,  multiple  modalities  can  be  used  to  maintain 
or  improve  competencies,  including  one  or  more  of  the  following: 

•  a  training  course  or  curriculum 

•  mentoring  or  other  on-the-job  training  opportunities,  such  as  ride-alongs 

•  shadowing  management  or  other  subject  matter  experts 

•  a  realistic  simulation  environment 

•  targeted  self-study  (e.g.,  technical  journals,  online  discussions,  or  topical  blogs) 

•  conference  attendance  and  participation 

•  academic  coursework  or  degree  programs 

2.1.4  Validate 

Validate  is  the  roadmap  activity  that  measures  whether  an  individual’s  training  and  development 
actions  have  addressed  his  or  her  competency  needs.  Validation  of  acquired  competencies  is 
achieved  by  conducting  a  performance-based  test  to  determine  if  an  individual  has  maintained  or 
improved  identified  competencies  through  his  or  her  actions.  It  defines  a  structured  approach  to 
measuring  knowledge  and  abilities  that  have  been  acquired.  Table  4  presents  the  key 
characteristics  for  the  validate  activity. 

Table  4:  Characteristics  of  Validate  Activity 


Dimension 

Description 

What 

•  a  measure  of  whether  actions  have  addressed  identified  competencies 

Why 

•  to  ensure  that  identified  competencies  have  been  adequately  maintained  or  improved 

Validation  methods  can  include 

•  quizzes 

•  certification  exams 

•  targeted  interviews  (by  experts) 

•  performance  on  a  simulation  or  other  learning  exercise 

•  observation  of  employee  demonstrating  what  was  learned 


For  further  discussion  of  competencies  and  training,  see  Handbook  for  Developing  Competency-Based  Training 
Programs  [Blank  1982], 
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Validation  focuses  on  the  knowledge  and  abilities  needed  to  perform  a  task.  Its  emphasis  on 
measuring  the  extent  to  which  knowledge  and  abilities  have  been  acquired  differs  from  that  of  the 
next  roadmap  activity,  test  readiness,  which  evaluates  the  application  of  knowledge  and  abilities 
in  an  actual  work  environment. 

2.1.5  Test  Readiness 

Often  overlooked  or  grouped  with  validation,  the  test  readiness  activity  of  the  roadmap  is  a  real- 
world  evaluation  of  whether  a  person  can  perform  a  specific  task  as  required.  People  bring  a  range 
of  knowledge  and  experiences  to  any  job  setting  or  task.  The  initial  assessment  and  subsequent 
validation  will  determine  an  individual’s  knowledge  of  and  experience  with  certain  competencies 
and  his  or  her  understanding  of  some  targeted  (often  highly  technical  or  organization-specific) 
competencies. 

However,  knowing  an  individual’s  current  proficiency  in  selected  competencies  is  insufficient  for 
predicting  that  individual’s  overall  readiness  to  perform  a  given  task.  An  individual  might  have 
related  knowledge  and  abilities  but  might  not  be  able  to  apply  them  in  a  real-world  setting.  The 
ability  to  test  an  individual’s  readiness  to  perform  a  task  is  an  essential  component  of  an  effective 
training  and  development  program.  Table  5  presents  the  key  characteristics  of  testing  an 
individual’s  readiness  to  perform  assigned  tasks. 

Table  5:  Characteristics  of  Test  Readiness  Activity 


Dimension 

Description 

What 

•  a  real-world  evaluation  of  whether  a  specific  task  can  be  performed  as  required 

Why 

•  to  ensure  that  competencies  can  be  appropriately  applied  to  tasks 

Methods  for  perfonning  readiness  testing  can  include 

•  real-world  scenario 

•  role-playing 

•  capstone  exercise 

•  real-world  simulation 

•  observation  of  real-world  task  performance 

Readiness  is  best  evaluated  using  a  multidimensional,  performance-based  evaluation  of  task 
performance.  The  multiple  dimensions  used  to  test  readiness  comprise  several  inputs,  including 

•  outcomes  from  assessments 

•  outcomes  from  evaluations 

•  interviews  with  supervisor(s) 

•  interim  job  performance  evaluations  (e.g.,  at  30  days  and  90  days  after  job  assignment) 

•  observation  by  supervisor  on  ability  to  perform  as  a  team  member  or  to  perform  a  specific 
job  function 

•  ability  to  explain  job  tasks  and  concepts  to  newer  staff 

More  experienced  individuals  may  assess  their  own  readiness  for  new  work  or  added 
responsibility  by  using  the  CLR  themselves  or  with  a  supervisor  or  another  experienced  co¬ 
worker  who  can  provide  objective  feedback. 
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It  is  vital  during  this  phase  to  understand  the  critical  importance  of  real  (as  opposed  to  realistic) 
work.  To  know  if  someone  can  actually  perform  all  of  the  activities  related  to  a  set  of  job 
requirements,  all  of  the  key  components  of  that  job  must  be  made  available.  For  example,  for 
someone  to  be  ready  to  complete  a  detailed  report,  he  or  she  must  understand  more  about  it  than 
how  to  fill  in  the  blanks,  such  as  how  the  report  is  communicated  to  others,  where  it  is  stored,  and 
who  may  be  using  the  report  over  what  period  of  time.  To  detennine  that  person’s  readiness,  the 
entire  task  in  its  real-world  context  must  be  presented  so  that  the  staff  member  can  demonstrate 
strengths  and  identify  areas  in  which  he  or  she  may  need  additional  development. 

Readiness  dimensions  are  often  tailored  to  the  job  requirements  and  specific  sets  of  competencies 
needed  to  both  perform  and  excel  at  that  job.  Sometimes  an  individual  may  develop  task 
performance  over  time.  In  other  instances,  depending  upon  the  role,  the  task  requires  performance 
excellence,  and  an  individual  is  deemed  either  ready  or  not  ready  to  perform. 

2.2  Foundational  Elements 

The  CLR  defines  a  foundational  element  as  an  entity  that  supports  the  execution  of  roadmap 
activities.  The  inclusion  of  foundational  elements  is  one  of  the  most  important  ways  in  which  the 
CLR  differs  from  other  models  and  approaches  to  many  training  and  development  programs, 
whose  moderate  success  in  achieving  desired  outcomes  may  be  due  in  part  to  their  exclusion  of 
such  elements.  These  critical,  enabling  elements  of  the  CLR  are  criteria  and  environment 
(sometimes  called  context). 

2.2.1  Criteria 

Criteria,  the  first  foundational  element  of  the  roadmap,  are  the  sets  of  technical  and  enabling 
competencies  that  define  the  requirements  for  performing  tasks.  Technical  competencies  are  the 
subset  of  knowledge  and  abilities  that  directly  affect  the  ability  to  perfonn  a  task.  For  example,  a 
technical  competency  for  a  cybersecurity  analyst  is  the  ability  to  use  an  intrusion  detection 
system.  For  a  project  manager,  the  ability  to  develop  a  schedule  is  a  technical  competency. 

In  contrast,  enabling  competencies  indirectly  support  the  completion  of  a  task.  Effective 
communication  is  an  example  of  an  enabling  competency.  For  example,  the  cybersecurity  analyst 
needs  to  communicate  information  about  possible  security  incidents  with  his  or  her  colleagues. 
Likewise,  the  project  manager  needs  to  communicate  with  his  or  her  team  when  preparing  and 
implementing  the  project’s  schedule  of  events. 

Criteria  establish  the  scope  of  performance  requirements  that  define  readiness  to  perform  a  task. 
Just  as  other  training  and  competency-based  programs  have  demonstrated,  our  research  indicates 
that  competencies  are  contextual.  They  work  best  when  aligned  with  a  role  and,  in  particular, 
when  the  specific  application  for  perfonning  the  role-based  functions  describes  the  competency  in 
terms  of  the  work  that  is  actually  done. 

Table  6  defines  the  characteristics  for  the  criteria  element  of  the  roadmap. 
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Table  6:  Characteristics  of  Criteria  Element 


Dimension  Description 

What  •  sets  of  technical  and  enabling  competencies  that  define  the  requirements  for  performing 

assigned  tasks 

•  can  also  be  broken  out  at  a  more  detailed  level  with  the  knowledge  and  skills  needed  to 
perform  the  activities  associated  with  the  competency;  could  also  include  the  tasks  performed 
as  part  of  the  competency 

Why  •  to  establish  the  scope  of  performance  requirements  that  define  readiness 


The  scope  of  performance  is  more  granular  than  it  appears  at  first  glance.  Specific  criteria  must  be 
understood  clearly,  both  in  depth  and  breadth.  And,  perhaps  most  importantly,  they  must  be 
specified  for  every  phase  or  activity  of  the  CLR  in  such  a  way  that  individual  growth  in 
knowledge  and  performance  can  be  measured  in  a  variety  of  ways,  over  time. 

The  CLR  can  be  applied  across  many  disciplines  in  part  because  it  can  incorporate  any  criteria, 
such  as  those  from  the 

•  National  Initiative  for  Cybersecurity  Education  (NICE)  cybersecurity  competencies  [NICE 

2011] 

•  OPM  Leadership  Competency  Framework  [OPM  2006] 

•  OPM  Competency  Model  for  Cybersecurity  [OPM  2011] 

•  Project  Management  competencies  from  the  Project  Management  Institute  [PMI  2002] 

2.2.2  Environment 

The  second  foundational  element  of  the  roadmap  is  the  environment,  which  includes  the 
processes,  culture,  and  context  that  influence  the  execution  of  the  roadmap  activities.  In  some 
instances,  conditions  within  the  environment  facilitate  or  enable  the  successful  completion  of 
roadmap  activities.  In  other  instances,  conditions  hinder  the  execution  of  the  roadmap  activities, 
acting  as  barriers  to  a  successful  training  and  development  program. 

The  roadmap’s  environment  element  gives  users  the  structure  and  support  they  need  to  work  in  a 
dynamic  organizational  setting.  The  environment  is  important  because  it  ensures  a  strong 
relationship  between  the  training  and  development  program  and  people’s  readiness  to  perform 
their  assigned  tasks.  Table  7  presents  the  key  characteristics  for  the  training  and  development 
program’s  environment. 

Table  1:  Characteristics  of  Environment  Element 

Dimension  Description 

What  •  processes,  culture,  and  context  that  influence  the  execution  of  the  competency  lifecycle 

activities 

Why  •  to  provide  the  structure  and  support  needed  to  work  in  a  dynamic  organizational  setting 

•  to  ensure  a  strong  relationship  between  training  and  development  goals  and  readiness  to 
perform  assigned  tasks 


Key  environmental  factors  in  the  success  of  a  training  and  development  program  can  include 

•  a  designated  training  coordinator  to  ensure  the  training  program  is  institutionalized,  updated, 
and  socialized 

•  clear  policies  and  processes  that  detail  staff  and  management  requirements  for  training  and 
development 
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•  outlined,  streamlined  processes  for  achieving  assigned  activities 

•  time  for  management  to  meet  with  each  employee  to  discuss  career  and  professional 
development  and  perform  a  yearly  assessment  of  the  needed  knowledge,  skills,  and  abilities 
for  performing  job  functions  satisfactorily 

•  time  for  staff  to  pursue  training  and  development  activities,  even  providing  training  as  a 
work  assignment 

•  a  centralized  tracking  system  to  allow  management  and  staff  to  track  training  plans  and 
accomplishments 

•  a  culture  of  training  and  education  within  the  organization  that  recognizes  the  importance  of 
developing  and  sustaining  competencies  and  encourages  such  pursuits  through  verbal 
communication  and  dedication  of  time  and  resources 

•  recognition  by  staff  and  management  that  training  and  development  is  more  than  just 
completing  yearly  compliance  modules  for  ethics,  security,  privacy,  and  other  such  practices 

2.3  Roadmap  Implementation  over  Time 

The  roadmap  can  be  used  to  define  an  individual’s  training  development  path  over  time.  Figure  2 

illustrates  this  process,  in  which  the  roadmap  establishes  an  individual’s  progression  from  novice 

to  expert  for  a  given  job  assignment. 


Novice  >  Intermediate  ^=>  Expert 


Assess 

Plan 

Acquire 

Validate 

Test  Readiness 

Assess 

Plan 

Acquire 

Validate 

Test  Readiness 

Assess 

Plan 

Acquire 

Validate 

Test  Readiness 

Novice  Criteria 

Intermediate  Criteria 

Expert  Criteria 

Environment 


Time 

Figure  2:  Competency  Lifecycle  Roadmap  (CLR):  Progression  over  Time 

Each  level  of  job  proficiency  (novice,  intermediate,  and  expert)  has  a  unique  set  of  criteria  that 
defines  the  requirements  for  performing  assigned  tasks  at  that  level.  The  environment  affects  all 
levels  of  job  proficiency  and  either  facilitates  or  hinders  an  individual’s  progression  over  time.  In 
addition,  the  core  roadmap  activities  (assess,  plan,  acquire,  validate,  and  test  readiness)  are 
performed  continuously  throughout  this  progression.  The  progression  across  levels  of  expertise 
defines  an  individual’s  training  and  development  path  within  an  organization,  providing  a 
roadmap  for  improving  an  individual’s  knowledge  and  abilities. 


CMU/SEI-2012-TN-020  |  10 


It  is  important  to  note  that  this  roadmap  is  not  intended  to  be  used  in  a  linear  fashion.  All  people 
have  some  areas  of  expertise,  some  areas  that  need  to  be  developed,  and  perhaps  some  areas  that 
are  outside  an  individual’s  interest  or  ability.  The  notion  of  readiness  may  be  iterative  and 
certainly  takes  into  account  criteria  at  all  levels,  from  novice  to  expert.  While  time  is  an  important 
factor,  it  is  only  one  indicator  of  the  growth  of  competency-based  readiness.  Other  indicators 
might  include  experience  or  collective  abilities  of  a  work  team. 
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3  Implementation  Approaches 


An  organization  can  use  the  CLR  instrument  as  a  guide  for  multiple  scenarios,  including 

•  building  an  initial  training  and  development  program 

•  benchmarking  its  training  and  development  program  and  identifying  gaps  and  areas  for 
improvement 

•  developing  curricula  or  training  programs 

•  helping  an  individual  or  team  set  personal  goals  related  to  a  specific  job  or  task 

This  section  of  the  technical  note  shows  how  the  CLR  can  be  used  effectively  for  each  use  case. 

3.1  Building  a  Training  and  Development  Program 

An  organization  can  use  the  seven  components  (five  activities  and  two  foundational  elements)  of 
the  CLR  as  a  guide  to  building  a  training  and  development  program.  The  CLR  focuses  not  on 
completion  of  curricula  or  courses,  but  rather  on  an  outcome  of  readiness,  which  may  entail 
expansion  of  the  training  and  development  program  to  ensure  its  long-term  success  and 
sustainability. 

To  build  a  training  and  development  program,  an  organization  might  perform  the  following  steps, 
adapted  from  the  work  of  Blank  [Blank  1982]  and  Gott  and  Lesgold  [Gott  2000], 

1 .  Choose  criteria  for  competencies  relevant  to  the  roles  in  the  organization  that  require 
training.  These  criteria  can  come  from  an  existing  set  of  competencies,  such  as  the  role- 
based  U.S.  federal  govermnent’s  cybersecurity  competencies  outlined  in  the  NICE,  or  they 
can  be  developed  in-house. 

a.  If  developing  criteria  in-house,  the  organization  can  use  a  role-based  scenario  technique 
that  gathers  information  from  staff  on  their  activities  and  the  skills  and  knowledge  they 
need. 

b.  This  data  can  then  be  synthesized  into  competencies  or  mapped  to  existing  sets  of 
competencies  to  determine  the  criteria. 

2.  Develop  strategies  for  assessing  staff  knowledge  and  skills.  A  preliminary,  self-administered 
inventory  of  mastered  skill  areas  can  help  ensure  that  staff  members  do  not  pursue  redundant 
training.  It  can  also  help  management  to  identify  projects  that  can  benefit  from  particular 
staff  skills. 

3.  Build  a  planning  step  into  the  training  and  development  process.  Individuals  in  the  program 
will  decide,  usually  based  on  discussions  with  their  manager,  which  competencies  they  need 
to  acquire,  which  might  need  to  be  refreshed,  and  which  require  only  sustainment  through 
professional  development.  Some  competencies  may  require  no  additional  development. 

a.  During  the  planning  step,  the  organization  should  determine  the  most  effective  methods 
for  providing  knowledge  and  skills  to  the  staff  member.  For  example,  some 
competencies,  such  as  adjusting  to  organizational  customer  needs,  may  be  learned  only 
through  mentoring  and  observation. 
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b.  The  individual  and  his  or  her  manager  should  work  together  to  develop  a  documented 
IDP  with  expected  competency  acquisition  and  completion  dates. 

4.  Provide  and  support  methods  of  competency  acquisition.  The  organization  should  provide 
ample  time  for  training  and  development  activities,  whether  they  are  courses,  self-paced 
computer-based  training  (CBT),  mentoring,  or  on-the-job  training.  The  organization  can  also 
pay  for  any  activities  that  charge  a  fee  such  as  formal  courses  or  university  programs.  It 
might  also  need  to  allow  trainees  time  away  from  current  duties  or  reduce  their  workload 
until  they  complete  the  activity.  All  supporting  information  and  components  of  this  step 
should  be  added  to  the  documented  IDP  established  in  step  3b. 

5.  Establish  methods  of  validating  that  the  completion  of  acquisition  methods  yields  the  desired 
outcome.  This  validation  can  be  measured  by  benchmarking  learning  objectives,  testing 
course  content  retention,  or  conducting  scenario  and  role-playing  activities  that  highlight 
what  was  learned. 

6.  Establish  a  method  of  readiness  testing  for  each  relevant  competency  or  competency  group. 
For  example,  in  the  opening  scenario  in  Section  1 :  Introduction,  real  readiness  testing  would 
involve  giving  the  staff  member  a  compromised  system,  with  no  instructions,  and  asking  him 
or  her  to  determine  what  happened  to  the  system  and  what  type  of  malicious  code  it  was 
infected  or  compromised  by.  Realistic  testing  would  match  the  staff  member  with  a  more 
experienced  partner  to  work  out  the  problem  as  they  would  on  a  real  task.  It  is  critical  to 
provide  both  the  problem  and  the  real  environment  or  context  in  which  the  work  problem  is 
likely  to  occur.  Without  this  readiness  testing  piece  of  the  training  and  development 
program,  organizations  cannot  adequately  assess  staff  members’  readiness  to  perform  the  set 
of  competencies  or  the  job  in  the  field  under  actual  working  conditions.  With  readiness 
testing,  organizations  can  assess  both  technical  and  enabling  skills,  including  complex 
judgment  and  deductive  and  inductive  reasoning,  problem-solving,  flexibility,  and  the  ability 
to  handle  unexpected  occurrences. 

7.  Establish  the  readiness  testing  and  competency  lifecycle  methodology  in  a  manner  that  is 
conducive  to  the  culture,  organizational  processes,  and  personality  of  the  business  unit  and 
parent  company.  The  assessment  activities  need  to  be  appropriate  to  the  organization  and 
must  have  no  negative  impact  on  day-to-day  operations. 

8.  Identify  and  develop  enabling  factors,  such  as  time,  funding,  and  other  resources,  to  ensure 
that  the  training  and  development  program  supports  the  entire  team  or  workforce  involved. 
Organizations  may  need  to  establish  an  institutionalized  process  for  the  training  program. 

The  process  should  include  standardized  mechanisms,  techniques,  and  templates  for 
assessment,  planning,  validation,  and  readiness  testing.  When  appropriate,  organizations 
(particularly  larger  and  more  complex  ones)  should  establish  support  mechanisms  (such  as 
policies,  procedures,  and  a  training  officer  position)  and  obtain  support  from  upper 
management  and  human  resources  for  implementing  the  training  and  development  program. 

The  steps  above  highlight  the  optimal  components  of  an  effective  program  for  developing  and 

maintaining  a  capable,  effective  workforce. 

3.2  Benchmarking  a  Training  and  Development  Plan  or  Program 

Organizations  can  use  the  CLR  to  benchmark  already  established  training  and  development 

programs.  Benchmarking  allows  an  organization  to  understand  its  program’s  current  strengths, 
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weaknesses,  gaps,  and  goals.  While  not  prescriptive,  such  benchmarking  can  shed  light  on 
unaddressed  areas,  particularly  those  regarding  competency  development  beyond  formal  training 
courses.  An  individual  (training  manager,  supervisor,  or  other  designated  professional)  or  group 
within  the  organization  can  self-administer  the  CLR  for  benchmarking,  or  an  external  party  can  be 
involved  to  reduce  bias. 

The  organization  can  evaluate  how  well  its  current  training  and  development  program  addresses 
each  of  the  CLR  components  (five  activities  and  two  foundational  elements).  This  benchmarking 
exercise  would  identify  any  gaps  in  the  program,  such  as  a  lack  of  readiness  testing  or  a  lack  of 
standard  performance  criteria  for  a  work  assigmnent.  This  method  can  also  help  to  identify  and 
prioritize  improvement  activities  to  address  any  gaps.  The  organization  might  determine  which 
CLR  components  need  to  be  incorporated  into  its  training  and  development  program  and  form 
plans  to  do  so.  If  an  organization  adds  a  component  of  the  CLR,  it  should  consider  the  relevant 
steps  and  issues  outlined  in  this  technical  note. 

CLR  users  can  also  benchmark  its  training  and  development  program  at  a  more  granular,  in-depth 
level  to  determine  how  well  each  component  is  performed.  This  more  focused  method  can 
identify  specific  strengths  and  weaknesses  and  establish  tailored  improvement  plans.  For  example, 
an  organization  might  discover  that  its  assessment  of  staff  knowledge  and  skills  did  not 
sufficiently  evaluate  strengths  and  gaps  in  essential  competency  areas. 

This  more  granular  benchmarking  can  be  done  in  numerous  ways,  depending  on  the  scope  of  the 
analysis  and  the  outcome  desired: 

•  self-assessment  by  a  training  officer  or  training  group 

•  focus  group  involving  a  representative  sample  of  staff,  management,  and  human  resources  or 
training  team  members 

•  survey  or  a  series  of  workshops  that  asks  staff  and  management  how  each  of  the  five  CLR 
activities  (assess,  plan,  acquire,  validate,  and  test  readiness)  are  performed 

Group  discussions  can  focus  on  the  training  and  development  program’s  competency  criteria  and 
their  enablers  or  constraints.  More  open-ended  discussions  might  generate  feedback  on  which 
areas  of  the  program  need  improvement.  These  discussions  can  also  provide  a  good  understanding 
of  the  staff  members’  general  like  or  dislike  of  the  existing  program,  which  may  shed  light  on 
how  well  they  use  it. 

3.3  Developing  Curricula  or  Training  Plans 

Organizations  can  use  the  CLR  to  guide  their  development  of  training  plans  and  curricula.  The 
CLR  can  be  especially  useful  when  developing  a  broad  curriculum,  which  entails  many  different 
competency  acquisition  methods  and  includes  tests  of  skill  readiness,  competency,  and  job 
function  level.  Using  the  components  of  the  CLR  as  a  model  can  help  build  not  just  a  set  of 
courses,  both  live  and  virtual,  but  also  a  synergistic  curriculum  that  focuses  on  the  practical 
experience  staff  need  to  complete  day-to-day  tasks. 

Using  the  CLR  to  help  develop  curricula  or  training  plans,  an  organization  can  choose  training 
activities  that  culminate  in  readiness  testing.  It  also  can  help  focus  planning  and  acquisition 
methods  on  those  requiring  observation,  demonstration,  role-play,  simulations,  or  shadowing  a 
more  experienced  staff  member.  This  approach  not  only  makes  the  training  curriculum  more 


CMU/SEI-2012-TN-020  |  14 


interesting,  but  it  can  also  improve  participants’  understanding  of  readiness  perfonnance 
requirements. 

3.4  Creating  an  Individual  Development  Plan 

To  create  an  IDP,  a  manager  or  supervisor  usually  works  with  a  staff  member  to  identify  the 
competencies  the  staff  member  should  acquire,  master,  refresh,  or  sustain  based  on  his  or  her  job 
function  or  role. 

If  possible,  the  staff  member  should  then  be  assessed  against  those  competencies,  based  on  years 
of  experience,  previous  job  performance,  certifications  in  the  knowledge  and  skill  domain,  or 
similar  parameters.  When  possible,  the  organization  should  conduct  more  granular  assessments  of 
current  competence  that  employ  a  skill-based  test  and  a  knowledge  test,  much  like  the  written  and 
driving  portions  of  driver’s  license  exams.  Such  an  assessment  provides  a  baseline  for  further 
measurement. 

To  develop  the  IDP,  the  organization  identifies  each  competency  and  its  corresponding  skills  and 
knowledge  areas.  Then  the  organization  documents  appropriate,  recommended  training  and 
development  opportunities  and  determines  the  methods  of  acquisition,  validation,  and  readiness 
testing. 

All  of  this  infonnation  is  documented  in  the  IDP,  so  both  the  staff  member  and  management  have 
an  explicit  understanding  and  agreement  about  competency  development  acquisition  and 
measurement  expectations.  The  organization  can  then  use  the  documented  IDP  as  an  individual 
roadmap  to  track  the  individuals’  progress  toward  completion  and  determine  if  and  how  he  or  she 
achieves  readiness. 
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4  Summary  and  Next  Steps 


The  CLR  provides  an  agile,  practical  approach  to  developing  and  managing  a  competency-based 
staff-readiness  program.  We  designed  the  CLR  around  the  idea  that  staff  members  at  all  levels  of 
expertise  require  periodic  readiness  assessments  for  both  existing  and  anticipated  work 
requirements.  The  CLR  provides  a  strategy  for  maintaining  and  enhancing  competence  over  time. 

The  enabling  competencies  are,  in  some  ways,  at  the  heart  of  this  approach  to  long-term  readiness 
development.  While  traditional  training  curricula  may  be  sufficient  to  maintain  and  enhance 
specific  skills,  they  are,  on  their  own,  insufficient  for  long-term  workforce  readiness.  New  skills 
must  be  integrated  with  existing  skills.  Organizations  and  their  individual  staff  members  must 
understand  when  older  knowledge  or  skills  are  sufficient  and  when  they  must  be  enhanced  or 
replaced  with  new  technologies  and  understandings. 

The  CLR  is  still  in  its  early  stages  of  development.  Our  next  steps  include  exploring  how  to 
describe  each  component  in  more  detail.  Next  steps  will  also  explore  how  readiness  testing  for 
more  esoteric  enabling  competencies  can  be  created  and  implemented.  Specific  next  steps  will 
include 

•  building  and  testing  assessment  tools  for  a  variety  of  readiness  requirements  at  both  the 
technical  and  leadership  levels 

•  piloting  applications  of  the  CLR  in  smaller  settings  with  individual  teams  or  work  groups 

•  using  the  CLR  in  a  variety  of  benchmarking  situations  that  might  include  training  and 
curriculum  design  as  well  as  mentoring  and  supervisory  programs 

•  documenting  scenario-building  techniques  so  they  can  be  used  for  both  assessment  and 
program  development 

•  documenting  the  role-analysis  methodology  for  identifying  competencies  for  different  roles 

•  developing  an  assessment  instrument  to  allow  organizations  to  benchmark  their  training  and 
development  programs  against  the  CLR  so  they  can  identify  areas  for  improvement 

•  exploring,  in  conjunction  with  other  groups  in  the  CERT  Program,  how  this  readiness 
approach  can  be  applied  to  a  team  rather  than  to  an  individual  as  described  in  this  technical 
note 

•  exploring  how  this  roadmap  can  be  applied  to  cybersecurity  training  and  development 
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